Night Owl 6

home *** CD-ROM | disk | FTP | other *** search

/ Night Owl 6 / Night Owl's Shareware - PDSI-006 - Night Owl Corp (1990).iso / 033a / ulp091b.zip / ULP.DOC < prev next >

Wrap

Text File | 1992-03-02 | 57KB | 1,090 lines

┌───────────────────┐ │ │ ║ UpLoadProcessor │ ╥ ╥ ╥ │ ║ │ ║ ║ ║ ╓──╖ │ ║ Version 0.91ß │ ║ ║ ║ ║ ║ │ ║ │ ╙───╜ ╨ ║──╜ │ ║ (c) Copyright 1992 - Stacy Smith │ ╨ │ ║ └───────────────────┘ ║ ════════════════════╝ Courtesy of: The Bloom Beacon-Picayune BBS (804) 525-9760 300-14,400 Baud (USRobotics V.32bis) FidoNet 1:276/112 Intelec Stacy Smith Route 6 Box 189 Forest, Virginia 24551 ┌────────────────────┐ │ 1. Introduction: │ └────────────────────┘ This system was born out of a need for a universal upload processor. There are many alternative systems available, but they are limited to the ZIP format and perhaps one or two others. Few are able to handle self-extracting archives. Most are limited in the number of levels of archive nesting allowed in a file to be tested. All require the use of a third-party duplicate file checking system if you want to screen your uploads for duplicates. I converted my BBS files over to the ARJ compression format, due to its superior compression ratio over PKZip and its features over LHA. I do not intend to debate the merits of one archive format over another, but due to my decision, the need for a universal upload processor became apparent, so off I went. While I was at it, I decided to incorporate other technologies, such as duplicate checking, archive format conversion, information lines, support for internal description files, etc., into a single package. This program is the result of my efforts to allow my BBS to handle any archive that my users can throw at it. ┌──────────────────────────────────────────────┐ │ 2. Features of the UpLoadProcessor System: │ └──────────────────────────────────────────────┘ ∙ Identifies and processes ARC, ARJ, HYP, LZH, PAK, ZIP, ZOO and GIF files, regardless of their file extensions (ideal for software distribution networks, like .SDN files). ∙ Identifies and processes ARJ, LZH, PAK and ZIP self-extracting (SFX) archives. ∙ Scans ARC, PAK, ZIP and ZIP SFX archives for DOS reserved keywords to prevent hacking by hex-editing. (ARJ and LHA are resistant to these type of hacking attempts). ∙ Detects ARJ security envelopes and ZIP authenticity verification (-AV) stamps, and can be left alone after processing (configurable). ∙ Selected uncompressed files uploaded can be processed and compressed using your default format. ∙ Removes known BBS ads from archives; includes a BBS ads maintenance utility so sysops can update their BBS ads databases in real time. ∙ Allows the use of up to 10 different archiving programs, all user- configurable. Any archiving program used that is not listed above will be identified using the file extension only, until it's signature is determined and incorporated into the program. ∙ Uses a recursive processing routine that will allow (theoretically) unlimited nested archives (the only limit is imposed by DOS). This routine has been tested to 5 levels deep as of this writing. ∙ Allows the use of up to 5 different file-checking programs, all user- configurable, for virus and trojan checking, etc. ∙ Allows the use of up to 5 different GIF file checking programs, all configurable, for integrity, etc.. ∙ User-definable disposition (rename or delete) of corrupted, duplicate or other archives (not virus-related). ∙ User-definable disposition (rename or delete) of virus-infected archives. ∙ Incorporates it's own CRC-32 duplicate checking system, as well as the associated database processing software. No more shelling out to (and registering!) third-party duplication-checking software, sacrificing speed and memory. The amount of duplication allowed is user-selectable. The internal duplicate checking routine can be disabled in favor of an alternative third-party system. This system is extremely fast with ARJ and ZIP archives, and it's database is much smaller than other systems. ∙ Converts all uploads into a default archive format of your choosing, or they may be re-archived in their original format (user-defined). Nested archives can also be converted to your default format, or re-archived using their original format. SFX archives can be archived using your default format, or optionally left alone after verification. ∙ Can utilize a user-defined time window (in months) for acceptance of new upload files, based on the most recent file date in the archive. ∙ Changes all archive file attributes to 'read/write' from 'read-only'. ∙ Supports the use of private and public upload directories. Moves files and upload descriptions from the private directory to the public directory. ∙ Honors the '/' identifier in the description marking the file as a private upload for the sysop by processing the file, but not making it public. ∙ Supports the use of DESC.SDI and FILE-ID.DIZ description files in an archive, user-configurable, for those not running PCBoard 14.5a+ and PCBDESC. ∙ Can optionally insert an archive information line in the file description that contains the number of files, total uncompressed size, oldest file date, and most recent file date. The information line format and content is defined by the user. ∙ Two modes of testing are available: normal mode, which fully unpacks the archive and tests each file individually; and fast mode, whichs scans a ZIP or ARJ archive directly for file CRCs and dates, and uses the archiving program's internal integrity testing. ∙ The online tester will accept a redirected ARJ or PKUNZIP archive listing file to verify the duplication and age limits before a user uploads the actual archive, saving him grief and gnashing of teeth. ∙ Includes a drop-in replacement utility for PCBoard's TESTFILE.EXE utility called ULPID, that returns an errorlevel based upon archive signature, not the file extension. ∙ User-selectable process logging to a disk file. ∙ Written mostly in C (and a little assembler) for optimal speed, using Borland C++ 3.0 and Turbo Assembler 3.0. ∙ Lifetime registration; pay ONCE and your registration number will work on all future versions! ┌─────────────────────────────────────────────────────┐ │ 3. Files Included in the ULP Distribution Archive │ └─────────────────────────────────────────────────────┘ ULP.EXE Event upload processing program. ULPTEST.EXE On-line testing program for use in PCBTEST.BAT. ULPDB.EXE CRC-32 database creation and maintenance program. ULPID.EXE Drop-in replacement for PCBoard's TESTFILE.EXE. ULPADS.EXE BBS ads database maintenance program. ULPADS.DB BBS ads database file. ULP.DOC This file. ULP.CFG Sample configuration file for the ULP system. HISTORY.TXT ULP revision history in reverse order. ULPBLT Archive verification bulletin for sysops to post. PCBTEST.BAT Sample PCBTEST.BAT file for using ULPTEST and ULPID. REGISTER.FRM Registration form for ULP. FILE_ID.DIZ Internal description file. When you unzip the distribution archive, you should see my PKZIP authenticity verification stamp, and a '-AV' after every file in the archive: # SSU301 The Bloom Beacon-Picayune BBS If there are any files missing or added, or the -AV stamp is missing, the archive has been tampered with. It would be advisable to call my BBS (listed at the top of this document) for the latest version of ULP. ┌───────────────────────────┐ │ 4. Program Requirements │ └───────────────────────────┘ To the best of my knowledge, this program will run on most any machine capable of running PCBoard 14.x. I have personally tested and developed ULP on the following machines, with no problems: 10 MHz 80286 computer (CompuAdd) with: ∙ PCBoard 14.5a/D ∙ 1 megabyte of motherboard memory with a 2 megabyte EMS card ∙ 115 megabytes of MFM hard disk space ∙ MS-DOS 5.0 with 4DOS 4.0 command interpreter ∙ EGA graphics subsystem ∙ NAS16550AFN serial port UARTs 33 MHz 80386 computer (Zeos) with: ∙ 8 megabytes of motherboard memory ∙ 130 megabytes of IDE hard disk space ∙ MS-DOS 5.0 with 4DOS 4.0 command interpreter ∙ DESQview 2.32 and QEMM 5.12 ∙ SVGA graphics subsystem ∙ 8250A/16450 serial port UARTs Other sysops that I have been in contact with have successfully implemented ULP on setups with CD-ROMs, networks (Ethernet and LANtastic) and other varying hardware. ULP has been developed and tested using the following archiving and file checking software packages: ARJ 2.30 (by Robert Jung) HYPER 2.5 (by P. Sawatzki and K. P. Nischke) LHA 2.13 (by Haruyasu Yoshizaki) LHarc 1.13c (by Haruyasu Yoshizaki) PAK 2.51 (by NoGate Consulting) PKPAK 3.61 (by PKWare) PKZIP 1.10 (by PKWare) PKZIP 1.93 Alpha (by PKWare) ZOO 2.10 (by Rahul Deshi) SCAN 8.3V86 (by McAfee Associates) GIFTEST 4.0 Beta (by Dave Navarro) The ULP system requires DOS 3.x or later, as it uses DOS SHARE-compatible file reads and writes, and uses the DOS PATH to find the archiving and other utilities. ULP's memory requirements are relatively small (about 128K or so, plus the disk buffer allocated), but all programs are spawned or shelled, which reduces the free memory for the program being executed. It would be a good idea to have as much free conventional memory as possible (ULP itself cannot use EMS or XMS memory, and really couldn't use it I wanted to), especially if you use the ARJ compression system, which requires in the neighborhood of 280K itself to run. ULPDB also should have as much conventional memory as possible, as it performs its sort in memory for optimal speed. If your duplication database is larger than 384KB (over 65,000 file CRCs), ULPDB will utilize a segmented sort algorithm, which is a little slower but capable of handling a database up to 6MB in size (over 1,000,000 file CRCs)! The memory sort will probably be fine for most systems, as 65,000 files would probably average out to be about 7,000 to 8,000 typical archives. If anyone creates a database more than 6MB in size, please contact me as I can implement another routine that could handle up to a 16MB database... A hard disk is required (and not very many BBSes don't have them) so it goes without saying (although I did anyway; someone is bound to ask...). Obviously, the 286+ version of these programs that is available to registered users will not run on a 8088, 8086 or 80186 based computer. So don't try it (I haven't, so I don't have a clue what will happen...probably a lock-up...). ┌───────────────────┐ │ 5. Registration │ └───────────────────┘ The ULP system is not free; nor is ULP is crippled to force registration. ULP is fully functional, and will always remain so. The only variation with the registered copies is no time delay and beg message. Why register? Besides a clean conscience, you will get two registration codes that will work for all future versions of ULP, and will remove the delay and message at the end of execution of each program. Why two codes? Well, the first code is for the shareware version of ULP, and the second code is for the 286+ version of ULP that is made available only to registered users of ULP. This version is slightly smaller and faster than the shareware, 086+ version. Also, registered users get a large time limit and unlimited downloading from my BBS, if they choose to do so... The registration fee for your unique codes is $15. Other variations are available; refer to the file REGISTER.FRM for all registration options. Please print the file REGISTER.FRM and fill it out. You can print out the form by issuing the following command from the DOS prompt: TYPE REGISTER.FRM > PRN ┌───────────────────────────────────────┐ │ 6. License, Warranty and Disclaimer │ └───────────────────────────────────────┘ I'll keep this part short and sweet, and dispense with the legal-ese: License: You are allowed to use ULP for 30 days, after which you must either register ULP or stop using it completely. ULP registration is a license for your use of ULP; I retain ownership of the software. A single registration applies to a single BBS system, regardless of the number of computers used in the system. If you run two or more distinct BBS systems on the same computer (with different names), you require two or more ULP registrations. Warranty: There isn't one. The only thing I'll guarantee is that ULP will take up disk space. Disclaimer: I'm not responsible for anything bad that happens. ULP works here, but I cannot be held responsible for it not working on your computer or doing any damage to hardware or software. If these aren't agreeable with you, then the best thing to do is delete ULP right now. I'll do my best to help any user (registered or not) that wants to use ULP, and I'll act on bug reports quickly, but I simply cannot and will not be responsible for anything bad, like lost data, disk crashes, or whatever else you can think of. ┌──────────────────────────────┐ │ 7. Principles of Operation │ └──────────────────────────────┘ Since the ULP system is made up of several interoperating programs, this overview and concept explanation should help you understand how ULP is designed to be used. ******************************************************************************* READ THIS SECTION VERY CAREFULLY, AS IT WILL MAKE LIFE MUCH EASIER!!! ******************************************************************************* I believe that all responsible BBS sysops verify all of their uploads prior to posting them, in order to protect both themselves and their users. ULP is designed with idea in mind. Most, if not all, sysops process uploads in one of two ways (listed with benefits and liabilities as I see them): 1) Make all uploads private, processing them during a system event. BENEFITS: ∙ Does not take up any on-line time on the user's part to process archives. ∙ Allows the conversion of all archives to a default format, so that the BBS archives are consistent. ∙ Allows the BBS to accept any archive format...face it, it's hard enough to get some of these weenies to upload, much less compress them the same way. LIABILITIES: ∙ Files are not available immediately for download. ∙ Does not catch duplicates or aged archives until after the user has uploaded them, and perhaps leads to abuse by clever (?) users. (It is assumed that these sysops still use the venerable 'PKUNZIP -T' in their PCBTEST.BAT...) 2) Process (test) each upload online after the user uploads them. BENEFITS: ∙ Catches duplicate, defective and aged archives while the user is online, denying him upload credits. ∙ Files are available immediately for download if they are not made private in the PCBoard setup. LIABILITIES: ∙ Takes up on-line time for a user, potentially adding to his long-distance phone bill, discouraging further uploading; this process is typically quite slow for large archives. ∙ The archive format cannot be converted (a limitation of PCBoard), leaving many different archive formats in your BBS directories. Because of this, many sysops force users to use a specific format, which may or may not be consistent with other BBS's the user calls. The ULP system takes the best of both of these methods of testing. ULP.EXE is an event-mode processor which will fully process every archive found in the source upload directory, including archive format conversion, and move the file and description to the destination directory. In the base setup, you MUST run ULP.EXE in your event to completely process all uploads to your system. ULPTEST.EXE is an on-line processor, with a twist: it doesn't recompress the archive. It does absolutely nothing more than testing; this will greatly reduce the amount of on-line time the user is forced to waste waiting for file processing. The conversion process is handled by ULP during the system event. Pay attention to this part: PCBoard normally has two upload directories for each conference: a private and a public directory. When PCBoard invokes PCBTEST.BAT, the upload is in the private directory. If the archive fails the testing, it will remain there. However, if it passes, one of two things will occur; if you have made all uploads private, the file will remain in the private directory; if you have not made uploads private, it will be moved to the public directory. There lies the rub: since ULPTEST.EXE only tests the archive, ULP.EXE must reprocess it (performing more elaborate tests and converting the archive format), and ULP.EXE is designed to move the file and description to a new location after testing, presumably to prevent users from getting to untested files. If you have made all uploads private, then setup and configuration of ULP is a snap: the source directory is the private upload directory, and the destination is the public directory. However, if you want to allow users to get to the public upload directory, then you must make a SECOND upload directory (in DLPATH.LST, just like another directory) for ULP.EXE to move files to. In this case, the source directory would be the public directory that PCBoard has tossed the files into after testing, and the desination directory is the new upload directory you have created in the DLPATH.LST file. To illustrate the operation: MAKE ALL UPLOADS PRIVATE │ ALL UPLOADS AVAILABLE AFTER TESTING ───────────────────────────────────┼───────────────────────────────────── 2 directories: C:\PRIVATE │ 3 directories: C:\PRIVATE C:\PUBLIC │ C:\PARTIAL │ C:\FULL │ User uploads a file, gets placed │ User uploads a file, gets placed in C:\PRIVATE by PCBoard. │ in C:\PRIVATE by PCBoard. │ ULPTEST.EXE tests it. │ ULPTEST.EXE tests it. │ PCBoard leaves file in C:\PRIVATE │ If it passes, PCBoard moves it to │ C:\PARTIAL; if it fails, PCBoard │ leaves it in C:\PRIVATE │ ULP.EXE reprocesses it and moves │ ULP.EXE reprocesses it and moves it it to C:\PUBLIC │ to C:\FULL ULPTEST.EXE also has two modes of operation: normal and fast. The normal mode decompresses the files, performs file, duplication and age checking, and then deletes the extracted files and returns to PCBoard. The fast mode DOES NOT decompress the file; it firsts peforms an archive integrity check, then scans ARJ and ZIP archives directly for duplicate and age checking, and then returns to PCBoard (if the archive is not ARJ or ZIP, then normal mode is invoked). In fast mode, file checking (viruses, etc.) is left for ULP to do (which is why the above discussion regarding private/public directories is important). ULPTEST.EXE will also accept a redirected ARJ or PKUNZIP listing text file named VERIFY.ULP as input to pre-verify an upload for a user, before the user actually spends his time uploading the file only to find out it won't pass the limits you set. ┌───────────────────┐ │ 8. Installation │ └───────────────────┘ GENERAL (REQUIRED!): ─────────────────── Make a subdirectory on your hard drive. For the purposes of this document, we'll call it "C:\ULP". Unarchive the ULP distribution archive into this subdirectory. You've more than likely already made it this far, if you're reading this file. <grin> The ULP system opens several files at once for various reasons. I would recommend that you have a minimum of FILES=40 in your system CONFIG.SYS file for a single-node system, since ULPTEST is run in conjunction with PCBoard. If you are running PCBoard under a network or a multitasking operating system, you should already have DOS's SHARE.EXE loaded. You must have SHARE loaded in order to take advantage of the file sharing and locking methods used by the ULP programs to prevent data loss. (If you are running a single-node system without a multitasker, SHARE is not needed). Edit the configuration file to suit your needs. Proper configuration will require you to refer to this section, the previous section and the section titled "Configuration", which has an in-depth explanation of each configuration parameter and its function. It would probably be wise to fire up your printer for this document... For this document, we'll assume the name of your configuration file is "ULP.CFG". Note that for any of the ULP programs, you can obtain a limited program syntax screen simply by executing the program with no arguments. FOR EVENT-MODE UPLOAD PROCESSING (REQUIRED!): ──────────────────────────────────────────── As a minimum setup, you MUST run ULP as an event-mode processor, as ULP.EXE handles all of the CRC database updating, archive conversion, file and description moving, archive information line computation, and other features. Further, ULP requires that you have 2 upload disk directories and 2 upload DIRxx files. They can be either a private/public setup or 2 public directories setup, as described below (and in the theory section): - PRIVATE/PUBLIC SETUP: This setup will not allow users to get to any archive that has not been fully processed. In PCBSETUP, under Main Board Configuration (and any other conferences you want ULP to process), make all uploads private. Define a private upload directory and a private DIRxx file (which should have already been done, since all sysop-only uploads are placed here by PCBoard). Also create a public upload directory and public DIRxx file. This is how I have my system setup. - TWO (2) PUBLIC DIRECTORY SETUP: This setup will allow users to download archives from both public upload directories; one directory has been tested, and the other has been fully tested and converted. In PCBSETUP, under Main Board Configuration (and any other conferences you want ULP to process), define a private upload directory and a private DIRxx file, and a public upload directory and a public DIRxx file (this is where PCBoard will place all new non-private uploads since you did not make them private). Also create a SECOND public upload directory and public DIRxx file, and add these to the DLPATH.LST, DLFILE and DLFILE.LST files. To process your uploads during your system event, add the following lines to your EVENT.SYS file (be sure to change the drive and path as necessary): C: CD \ULP ULP ULP.CFG ULP will lock the CRC-32 database, preventing any other program from accessing them (ULPTEST, specifically). Therefore, I strongly suggest you have all nodes down when running ULP. FOR THE INTEGRATED CRC-32 DUPLICATION CHECKER (HIGHLY RECOMMENDED): ─────────────────────────────────────────────────────────────────── Initially, you must create the CRC database from scratch. If you have mostly ZIP and ARJ files, then this should be very quick (on the order of 10 minutes per 1000 archives for my 10MHz AT clone). After you edit the configuration file, go to the ULP directory and this command will create and compile your database of your current downloadable file base: ULPDB ULP.CFG ADD:@C:\PCB\GEN\DLPATH.LST The '@' symbol (a '!' may also be used) indicates to ULPDB that the argument is a list file, not a directory. The list file must have one path per line, and each path must end with a trailing backslash. In general, you may simply use your PCBoard DLPATH.LST file, unless you are indexing your downloads. If you would like to include any off-line files in your CRC database (or have to add files originally not included in the database) you can temporarily copy your offline files to a temp directory (for the sake of argumentation, let's call it "C:\TEMP\"; note the trailing backslash). You can then add and compile them into the CRC database with the following command (no list symbol!): ULPDB ULP.CFG ADD:C:\TEMP\ Note that if you want to add a CD-ROM, or any other write-protected storage system, you will have to use the WORK_DIR configuration parameter for any GENERATE or ADD function on this storage media. After you've added the offline files, just delete them from the disk, and if someone uploads a file that you already have, it will be rejected. Once you have your database built, you should also perform maintenance on the CRC database files once per day. This will compile any new CRCs from the day's uploads into the main database, and remove any added by ULPTEST. This is not required to be done every day, but it will speed the duplicate checking in ULP and ULPTEST. To perform database maintenance, add the following line to your PCBoard EVENT.SYS after the event-mode processor has run: C: ──────┐ These lines are repeated CD \ULP ├───── for clarity. ULP ULP.CFG ──────┘ ULPDB ULP.CFG COMPILE ULPDB also locks the CRC-32 database, preventing any other program from accessing them. I strongly suggest you have all nodes down when running ULP. FOR THE ARCHIVE IDENTIFICATION PROGRAM (OPTIONAL): ────────────────────────────────────────────────── NOTE: If you are unfamiliar with the operation of PCBTEST.BAT, you will need to refer to the PCBoard manual on pages 200-202 for an explanation of PCBTEST.BAT and TESTFILE.EXE ULPID is the ULP system's file identification program, which is a drop-in replacement of PCBoard's TESTFILE.EXE. To install it, simply replace the 'TESTFILE' in your PCBTEST.BAT with 'ULPID ULP.CFG'. This is not required for ULPTEST to work; it returns an errorlevel in the same manner as TESTFILE, but uses the same archive signature routine that ULP uses to identify an archive. For example, if your original line was: TESTFILE %1 ARC ARJ LZH PAK ZIP ZOO GIF then replace it with: ULPID ULP.CFG %1 ARC ARJ LZH PAK ZIP ZOO GIF ULP In order to take advantage of the upload pre-verification abilities of ULPTEST, you must add the extension 'ULP' to the ULPID extension list, or to the TESTFILE extension list if you choose not to use ULPID. This will allow the file 'VERIFY.ULP' to pass through the filter created by TESTFILE and ULPID so that ULPTEST may scan it. DO NOT add 'EXE' to your extension list. ULPID will automatically determine which format was used to create an SFX, and will exit with the appropriate errorlevel. FOR THE ONLINE UPLOAD TESTING (OPTIONAL): ───────────────────────────────────────── Online testing of archives can greatly reduce the amount of garbage your BBS can pick up from user's uploading defective archives, ancient files, and duplicates. The theory that ULPTEST uses is to simply check the archive AS FAST AS IT CAN for archive integrity, duplication, and age, and then return the user to greater and better things. ULP.EXE will handle everything else during the system event. To use ULPTEST.EXE for on-line testing of archives, use the following command line in PCBTEST.BAT (refer to the sample PCBTEST.BAT included in the ULP distribution archive, and the PCBoard manual, to get the general logic of PCBTEST.BAT's operation): ULPTEST ULP.CFG 1 %1 %2 The '1' is the node number (for a single node system, it doesn't matter as long as the value is between 1 and 256, inclusive). For multi-node systems, use the %PCBNODE% environment variable in its place (as long as you aren't using the /NOENV switch in PCBoard): ULPTEST ULP.CFG %PCBNODE% %1 %2 ULPTEST will return a DOS errorlevel of 0 if the archive is OK, and an errorlevel of 1-9 if there was an error (depending on the error). This allows the PCBTEST.BAT batch file to branch based upon the results of the ULPTEST. The appendix to this document lists all of the DOS errorlevels set by various ULP programs. ULPTEST also creates the requires PCBPASS.TXT and PCBFAIL.TXT for PCBoard to determine if the archive passed or not. ┌────────────────────┐ │ 9. Configuration │ └────────────────────┘ The sample configuration file included in the distribution archive is heavily commented, but some additional information is provided below: GENERAL PARAMETERS: ─────────────────── REG_CODE The registration code EXACTLY as provided to you on your registration letter. If this is an unregistered version, comment out this line. Note that this is case-sensitive, as the serial number, BBS name and sysop name are all encrypted into this code. BBS_NAME The BBS name EXACTLY as provided to you on your registration letter. If this is an unregistered version, comment out this lines. Note that this is case-sensitive! SYSOP_NAME The sysop name EXACTLY as provided to you on your registration letter. If this is an unregistered version, comment out this line. You get the picture by now... DISK_BUFFER The size of the disk I/O buffer you wish to allocate in bytes (valid values are between 512 bytes and 32768 bytes). Optimal performance is obtained with a value that is a multiple of 2048 bytes, since this is the size of a typical hard disk cluster. 4096 seems to be a good number for me, although a larger size will improve performance somewhat. If you find yourself running out of memory when executing archivers or file checking programs, reduce this number to increase your available memory. If no value is specified, the disk buffer will default to 4096 bytes. LOG_FILE All of the ULP programs with the exception of ULPID will write process and error information to a log file, if desired. This is the path/filename of the log file for ULP to record its activities. Comment out if you don't want any disk logging. LOG_MODE If you want ULP and ULPTEST to record their statistics and finding for each archive processed to the log file, use the VERBOSE keyword. If you only want errors and summary statistics logged to disk, use the TERSE keyword or comment the line out. VERBOSE mode will quickly produce a large log file, but gives much insight into what ULP is doing and why a particular upload failed. INT_DESC ULP can extract and insert FILE_ID.DIZ and DESC.SDI internal description files into the DIRxx files. If you want ULP to insert internal description files, type YES; if not, type NO or comment the line out. WORK_DIR If you want ULPDB to use a specific directory for decompressing archives while generating the database, place the pathname (including the trailing backslash). This directory must exist! If you do not desire to use a specific directory, comment it out (ULPDB will extract at its current location on disk). I would not recommend using a RAM disk, unless you are sure that extracted archives will fit on the disk. If an archive will not fit onto a RAM disk, ULPDB will flag it as a bad archive in the log file and not record its CRCs. The primary purpose for this purpose is to allow sysops with CD-ROM, WORM or other write-protected media to add those archives to the CRC-32 database. ERR_DISP This defines how ULP dispositions archives found to have an error other than a file check failure. These kind of errors take the form of ZIP CRCs, etc., and are not potentially system damaging. Under some conditions, these archives can be restored. If you want ULP to delete the archive and description, enter DELETE; if you want ULP to rename the archive to a .ERR extension, type RENAME or comment the line out. VIRUS_DISP This defines how ULP dispositions archives that fail a file check, such as a virus scan. These errors are (obviously) potentially system-damaging. (I wouldn't keep them if I were you.) If you want ULP to delete the archive and description, type DELETE; if you want ULP to rename the archive to a .VRS extension, type RENAME or comment the line out. SCREEN_ARC If you want ULP and ULPTEST to screen ZIP archives for DOS reserved keywords prior to unpacking them, type YES or comment the line out. Otherwise, type NO. This will prevent someone from hanging your system by hex-editing an ARJ or ZIP archive and replacing a filename with a DOS reserved device keywords (such as COMn, LPTn, PRN, CLOCK$, etc.). UPLOAD DIRECTORIES: ─────────────────── Each pair of upload directories (one source, and one destination), is denoted by UL_DIR x, where x is a sequential number. Up to 10 directory pairs can be processed by ULP. The next 6 keywords are REQUIRED after the UL_DIR statement; do not comment any of them out or insert other parameters or comments in between them: UL_DIR = denotes the beginning of a pair of upload directories definition. SOURCE_PATH = the directory where the uploads are placed by PCBoard. This must include the trailing backslash! NOTE: if you are using the 2 public directory setup, this parameter will be the INITIAL public upload directory; the same goes for the next two parameters). SOURCE_LIST = the source upload directory list file that contains the archive descriptions, including the path. SOURCE_HDR = the number of lines to ignore at the beginning of the source upload directory list file (for headers, detachable header filespecs, etc.) DEST_PATH = the destination directory where the uploads are placed by ULP after successful processing. This must include the trailing backslash! DEST_LIST = the upload directory list file that contains the archive descriptions, including the path. ULP will append this file with the descriptions of successfully processed files. DEST_HDR = the number of lines to ignore at the beginning of the upload directory list file (for headers, detachable header filespecs, etc.) AGE-LIMIT CHECKING: ─────────────────── AGE_CHK If you want ULP to fail an archive based on the date of its most recent file, type YES; if not, type NO or comment the line out. AGE_LIMIT The maximum age in months, that you will accept for an upload. (If the ERR_DISP is set to RENAME, it will be renamed to a .ERR extension). ARCHIVING CONTROL FLAGS: ──────────────────────── ARC_CVT If you want ULP to convert the archive to a default archiving format of your choice, type YES; if not, type NO or comment the line out. This only affects the 'outside' level of the archive. This flag has no affect on the conversion of internal, nested archives. ARC_NEST_CVT If you want ULP to convert any internal archives found inside the archive to your default archiving format, type YES; if not, type NO or comment the line out, and they will be rearchived using the original archiving method (using the original extension, even if it isn't that archiver's default extension). The benefit of converting archives is it may save some disk space; on the down side, it will potentially hose automatic installation programs/batch files that may be contained within the archive. SFX_CVT If you want ULP to convert a self-extracting (SFX) archive to your default archiving format, type YES; if not, type NO or comment the line out. If NO is selected, then the archive will be processed, and then left alone if it passes (NOT re-archived!). The reasoning for this is the LHA autostart batch files and other similar features; they are usually used for novice users and for automatic installation. By converting SFX archives, this may save some disk space, but can cause inconveniences. For example, if you don't have PKZ110.EXE on your BBS, a novice user has no way of obtaining the PKZIP utilities from your BBS. * NOTE: If there is a BBS ad in a SFX archive, and SFX_CVT is set to 'no', the ad will remain in the archive. KEEP_AV If you want ULP to keep ZIP archives with authenticity verification stamps (-AV) regardless of the other archiving control flags, then set this to 'yes', or comment it out. If you set this flag to 'no', then the archive will be recompressed per the other control flags. * NOTE: If there is a BBS ad in a -AV stamped archive, and KEEP_AV is set to 'yes', the ad will remain in the archive. KEEP_ENV If you want ULP to keep ARJ archives with security envelopes (similar to a ZIP -AV) regardless of the other archiving control flags, then set this to 'yes' or comment it out. If you set this flag to 'no', then the archive will be recompressed per the other control flags. * NOTE: If there is a BBS ad in a secured archive, and KEEP_ENV is set to 'yes', the ad will remain in the archive. ARCHIVER DEFINITIONS: ───────────────────── Each archiver is denoted by ARCHIVER x, where x is a sequential number. Up to 10 archivers can be executed by ULP. ULP is capable of automatically detecting ARC, ARJ, HYP, LZH, PAK, ZIP and ZOO archives. The next 6 keywords are required after the ARCHIVER statement; do not comment any of them out or insert other parameters or comments in between them. I would also advise against changing the command options I have set, unless you have need to do so. Some are there to ensure uninterrupted processing (e.g. /m1 in LHA). Any other archiver can be added, if desired; the only requirement is that the archiver must return a DOS errorlevel so that ULP can determine if it ran properly. If you do not explicitly define the path to your file checking program(s), it must in a directory included in the DOS PATH environment variable. * NOTE: The ARJ and ZIP formats MUST be included for ULP to function properly! ARCHIVER = denotes the beginning of an archiving program definition. ARC_EXT = the extension for this particular archiving format. ARC_PRIMARY = if this is the default archiving format you use on your BBS, type YES; if not, type NO. ARC_PACK = the command line to pack archives, including all command-line switches desired. Place the variable @ARCHIVE@ where the filename is to be inserted in the command line and @FILES@ where the filespec is to be inserted. If you want a comment to be inserted, you must include it in the command-line switches. This cannot be a batch file! (DOS re-direction is allowed) ARC_UNPACK = the command line to unpack archives, including all command-line switches desired. Place the variable @ARCHIVE@ where the filename is to be inserted in the command line. This cannot be a batch file! (No DOS re-direction allowed) ARC_TEST = the command line to test archives, including all command-line switches desired. Place the variable @ARCHIVE@ where the filename is to be inserted in the command line. This cannot be a batch file! (No DOS re-direction allowed) ARC_ERR = the errorlevel that the archiver returns upon success. This information should be available in the program's documentation. UNCOMPRESSED FILES TO PROCESS: ────────────────────────────── PROC_EXT ULP can process selected uncompressed uploads, using this option. Up to 10 uncompressed file types can be specified by extension. Do not use any compression format extensions (ARJ, ZIP, etc.)! Any files that are uploaded with one of these extensions and could not be identified by signature will be file-checked, duplicate-checked, and compressed using your default archiving method. If you do not want to use this feature, simply remove or comment out these lines. It probably is not a good idea to have EXE as an extension to compress, since ULP is not able to detect ARC, HYP or ZOO self-extractors, and they would be packed as an uncompressed archive. FILE-CHECKING (VIRUS SCANNING, ETC.): ───────────────────────────────────── Each file checker is denoted by FILE_CHECKER x, where x is a sequential number. Up to 10 file checking programs can be executed by ULP. The next 2 keywords are required after the FILE_CHECKER statement; do not comment them them out or insert other parameters or comments in between them. Any kind of file checker may be used, such as a virus checker, or a 3rd party CRC duplication checker, if desired. If you do not explicitly define the path to your file checking program(s), it must in a directory included in the DOS PATH environment variable. The only requirement is that it must return a DOS errorlevel, otherwise ULP has no clue if there was an error or not. FILE_CHECKER = denotes the beginning of a file-checker definition. FILE_CHK_CMD = the command line to execute each file checker, including all command-line switches. Place the variable @FILES@ where the filespec is to be placed on the command line. This cannot be a batch file! (No DOS re-direction allowed) FILE_CHK_ERR = the errorlevel that the file checking program returns upon success. This information should be available in the program's documentation. GIF FILE CHECKING ───────────────── Each GIF file checker is denoted by GIF_CHECKER x, where x is a sequential number. Up to 5 GIF file checking program(s) can be executed by ULP. In addition, do not use the GIF file checking program's internal duplicate checking or file handling abilities, if any. ULP and ULPTEST will take care of these operations. The next 2 keywords are required after the GIF_CHECKER statement; do not comment them out or insert other parameters or comments in between them. GIF_CHECKER = denotes a new file checking/virus program definition. GIF_CHK_CMD = the command line to execute each GIF file checker, including all command-line switches. Place the variable @FILES@ where the filespec is to be placed on the command line. This cannot be a batch file! (No DOS re-direction allowed) GIF_CHK_ERR = the errorlevel that the GIF file checking program returns upon success. This information should be available in the program's documentation. ARCHIVE INFORMATION LINE: ───────────────────────── INFO_LINE ULP can add a single line of archive information for the benefit of your users. The format is completely defined by you. If you want ULP to add a single line to the archive description, type YES; if not, type NO or comment the line out. Note that the statistics generated by ULP will not include any FILE_ID.DIZ or DESC.SDI files found in the archive. INFO_INDENT The column where the | marker goes in your directory listing setup. The standard PCBoard setup indents to column 32. INFO_FORMAT The format that you would like for the information line (do not include the '|' character or the space; ULP will add those). This format is completely configuration by the user; any ASCII character can be used. To include specific information compiled by ULP, just add the variables. The legal variables are (must be in all caps): @#@ = Number of files in the archive (and nested archives) @NEWEST@ = Date of the newest file in the archive @OLDEST@ = Date of the oldest file in the archive @BYTES@ = The total number of bytes (uncompressed) in the archive ULPTEST TESTING: ──────────────── TEST_MODE This defines how ULPTEST processes an archive. NORMAL indicates a full test is to be performed, identical to the ULP event processing except that the archive formats are not converted or recompressed (this is a limitation of PCBoard). FAST indicates a fast test in which the archive is scanned for archive integrity, duplicates, and age without unpacking the archive (ARJ and ZIP files only). If this mode is used, it should be held for later processing by ULP so that it can be file-checked for viruses, etc. Also, FAST mode can only 'see' one level deep in the archive. If you have a slow computer and/or hard disk, FAST mode would be preferable for your users when large archives are uploaded. If you comment this line out, NORMAL mode is assumed. I recommend you use NORMAL mode, and then set a limit with the next parameter for when FAST mode will be utilized. TEST_LIMIT If NORMAL mode is used, the archive's file size will first be compared against this limit (in kilobytes). If it is larger than this number of kilobytes, it will be forced into FAST mode to speed the process and prevent wasting the user's online time. Entering 0, or commenting the line out will disable this feature, resulting in NORMAL mode being used at all times. If TEST_MODE is set to FAST, this parameter has no effect. For my system, a good value is 200 kilobytes, as anything much bigger than this takes quite awhile to decompress, test, and delete the files extracted. BBS ADS FILE REMOVAL: ───────────────────── ADS_CHK If you want ULP to remove known BBS ads from new uploads during processing, type YES; if not, type NO or comment the line out. ADS_DB The directory path and name for the BBS ads database. This database can be updated by the sysop by using the ULPADS utility (see section 11 of this document), freeing the sysop from being reliant on me to maintain the ads database. If you find a large number of new BBS ad files, please upload them to my board for inclusion into my master ads database. CRC-32 DUPLICATE FILE CHECKING: ─────────────────────────────── DUPE_CHK If you want ULP to use its internal CRC-32 duplication checking, type YES; if not, type NO or comment the line out. If you want to use a third-party duplication checker, such as ZDCS, you can add it as a file checker (above), providing it returns DOS errorlevels so ULP can detect its success/failure. I would strongly suggest using ULP's, as it is fast and compact, not to mention integrated, eliminating the time to shell to disk for each file in each archive to be tested. DUPE_LIMIT The maximum percentage of duplication allowed for a file to pass. Anything larger will fail (0 will allow no duplications at all, while 100 will allow everything to pass). I would suggest this number be set to at least 50, as some program authors don't modify every file, especially with maintenance releases. DUPE_DB The path and name for the CRC-32 database file. It's a good idea to keep the database, index, and new data files all together, just to make things easier. DUPE_IDX The path and name for the CRC-32 database file index. ┌────────────────────────┐ │ 10. Manual Operation │ └────────────────────────┘ While the ULP system is mostly automatic, there are occasions where some operations may have to be done manually. During the course of operation, ULP will rename archives that have been found to be defective in some manner according to the following convention: .UNK Unknown archive format .DOS DOS reserved keyword found in archive .ERR Error occurred while unpacking archive (archive integrity failure) .VRS Error found while file checking archive file (virus, etc.) .DUP Excessive duplicate files contained in archive .PCK Error occurred while repacking archive file .AGE Age limit exceeded by archive file .BAD Error found while testing GIF file I would enter these extensions in your UPSEC file, as ULP will automatically pass over any file with these extensions, presuming that they are defective archives. Doing so will prevent a user from uploading a file with one of these "reserved" extensions. If you feel that these files are acceptable after reviewing them, you can force them to be accepted by adding the command-line switch OVERRIDE to the ULP command line, e.g.: ULP ULP.CFG OVERRIDE This will accept the archives regardless of duplication or age limits. It will not override unpacking, packing, integrity and virus errors, however, for the safety of your board and your users. ┌──────────────────────────┐ │ 11. BBS Ads and ULPADS │ └──────────────────────────┘ The ULP system includes a BBS ad removal feature based on CRC-32 calculation of the file contents and other data. In this fashion, ULP can detect a known ad file despite the file name and date. In order for sysops to be able to 'keep up' with new ads produced by the weenie sysops who insert the @!&*#%$ things, I have included a utility called ULPADS that can scan a BBS ad file, and update the BBS ads with it's information. Don't worry about duplication, as part of the process is to purge duplicate BBS ad info. The command-line syntax for the ULPADS.EXE program is simply: ULPADS DATABASE.AD BBS.AD where 'DATABASE.AD' is the full path and filename for the BBS ads database, and 'BBS.AD' is the path and filename for the BBS ad file to be added to the ads database. I would greatly appreciate your uploading of any new BBS ad files that you may collect over time to my BBS so I can update the master listing that I include with the ULP distribution archive. Please refer to the top of this document for my BBS number. ┌───────────────┐ │ 12. Support │ └───────────────┘ If you require support for ULP, I can be contacted by any of the following means: ∙ Fidonet Netmail addressed to Stacy Smith, node 1:276/112 ∙ Intelec Shareware conference ∙ Intelec PCBoard conference ∙ Salt Air (CDC would probably prefer the Sysop Chatter conference) ∙ The ULP Support conference on my BBS (listed at the top of this document). ┌─────────────────────────┐ │ 13. The Future of ULP │ └─────────────────────────┘ ULP will be supported as long as I'm in the BBSing business (which will be quite a while...once it's in your blood, you can never shake it <grin>). The ULP system will be rapidly expanding it's features so it will be your first choice in BBS upload processors. Some current plans: ∙ Incorporate code to handle archives containing explicit paths. ∙ Write a configuration program to ease the configuration process. ∙ Add the ability to preprocess files prior to file checking them; for example, decompress executables that have been PKLite'd. ∙ Support other BBS software directory listing formats. If you have any other suggestions, or want other archivers supported, please contact me by U.S. snail-mail or on my BBS at the number at the top of this document. Thanks for giving ULP a try! ┌────────────────────────────────┐ │ Appendix A: DOS Errorlevels │ └────────────────────────────────┘ The errorlevels returned to DOS by the ULP system programs are consistent among all of the programs (primarily with the error conditions); the following is a list of the errorlevels than can be returned by the ULP programs, along with which programs return the codes: 0 Successful execution (ULPTEST, ULPDB, ULPADS) 0 Successful execution, archives were processed (ULP) 1 Successful execution, nothing to do (ULP) 1 Unknown archive format (ULPTEST) 2 DOS reserved keyword found in archive (ULPTEST) 3 Error unpacking archive (archive integrity) (ULPTEST) 4 Error file checking archive files (virus, etc.) (ULPTEST) 5 Error duplicate checking archive files (ULPTEST) 7 Age limit exceeded by archive files (ULPTEST) 9 Bad GIF file (ULPTEST) 1-97 Matched extension number in list (ULPID) 98 Unmatched archive type (ULPID) 99 Help screen (executing the program with no or an insufficient number of arguments) (All) 100 Could not close all open files (ULP, ULPTEST, ULPDB) 101 Unknown command line parameter (All) 102 Config file not found (All) 103 Invalid config file format (All) 104 Excessive number of command line parameters defined for file checker (maximum number of parameters is 10) (ULP) 105 Excessive number of command line parameters defined for archiver (max is 10) (ULP, ULPTEST, ULPDB) 106 Excessive number of words defined in archive information line (max is 10) (ULP) 107 No primary archiver defined (ULP) 108 Invalid disk buffer size (All) 109 Upload directory list file(s) not found (ULP, ULPTEST) 110 Unable to allocate heap memory (All) 111 Unable to open or lock disk logging file (ULP, ULPTEST, ULPDB) 112 Invalid registration code or attempting to run 286+ version without registering (ULP, ULPTEST, ULPDB) 113 Unable to open/lock CRC database file(s) (ULP, ULPTEST, ULPDB) 114 Unable to open BBS ads database file (ULP, ULPADS) 115 Invalid filename provided by user on command-line (ULPTEST, ULPID) 116 Invalid path provided by user on command-line (ULPDB) 117 Invalid node number (valid values are 1 to 256) (ULPTEST) 118 CRC database usage not enabled (configuration file parameter DUPE_CHK is set to 'no') (ULPDB) 120 Invalid or old database format. (All) 200 Undefined error (internal to ULP system program(s))